Show browser screenshots in the chat transcript - #303
Open
frenchie4111 wants to merge 2 commits into
Open
Conversation
Screenshots the agent takes with ness-control browser tools were being dropped on the floor in Chat tabs — the tool_result extractor only kept text parts, so you could see that a screenshot happened but never what it showed. Images are spilled to the existing on-disk attachment store and only the path travels in slice state, since every state event fans out to every connected client and a PNG is megabytes of base64. Filenames are content -addressed so resuming a session (which replays the whole transcript through the extractor) doesn't leak a fresh copy of every screenshot. The tool card and its enclosing group both auto-expand when a result carries an image, so the screenshot isn't buried behind two chevrons. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Every expanded generic tool card rendered its input three times: the collapsed subtitle, a raw JSON.stringify pre, and then ArgsBlock — which shows the same values parsed and already has a "show raw" toggle for the identical JSON. The pre was the ugliest of the three and the only one with no reason to exist. Barely noticeable before, since these cards were collapsed by default. Auto-expanding them for screenshots put it on screen. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When the agent takes a screenshot with the ness-control browser tools in a Chat tab, the transcript now shows what it saw. Previously the tool_result extractor kept only text parts, so the image was dropped on the floor — you could see that a screenshot happened, but never what was in it.
How the image gets there. Tool-result images are spilled to the existing on-disk attachment store and only the path travels in slice state. A PNG screenshot is megabytes of base64 and every state event fans out to every connected client, so putting the bytes in the reducer wasn't an option. The renderer reads them back lazily through the same
jsonClaude:readAttachmentImagepath that pasted attachments already use.Filenames are content-addressed (sha256), because resuming a session replays the entire transcript back through the extractor. With uuid names that would leak a fresh copy of every screenshot in the history on every resume.
Two expansion fixes, so the screenshot isn't buried: the tool card auto-expands when its result carries an image, and so does the enclosing
ToolGroup. Both track the user's click as its own tri-state (boolean | null) rather than reverting via an effect — the tool_result arrives after the card mounts, so auseState(autoExpand)initial value would capturefalse; and sincehasImagesnever goes back to false, an effect-driven revert would make a screenshot group impossible to collapse.The second commit removes an unrelated wart it exposed: every expanded generic tool card was rendering its input three times — the collapsed subtitle, a raw
JSON.stringify<pre>, and thenArgsBlock, which shows the same values parsed and already has ashow rawtoggle for the identical JSON. The<pre>was redundant with both. Barely noticeable while these cards were collapsed by default; auto-expanding them put it on screen.Not covered
Only screenshots produce an image.
click_tab/type_tabreturn plain text, so nothing visual renders for them — seeing where a click landed would need Ness to auto-capture after each interaction, which is a bigger and more invasive change.Test plan
npm run typecheck,npx electron-vite build,npm run build:web— all cleannpx vitest run src/renderer— 40 files / 504 tests passtoolResultAttachedcarries image refs onto the blockJSON.stringify(block), bytes round-trip, and extracting the same image twice returns the same pathnpm run dev: opened a Chat tab, had the agent screenshot example.com, confirmed the thumbnail renders, the card and group auto-expand while sibling cards stay collapsed, and click-to-zoom + Escape workshow rawstill reaches the same JSON🤖 Generated with Claude Code